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DECISION MAKING IN CLASSIFICATION PROBLEMS 



Field of the Invention 

The invention relates to decision making in classification problems and relates 
5 particularly, though not exclusively, to improved methods of classification in decision fusion 
applications. 

Background of the Invention 

Decision fusion is a widely used technique for several kinds of classification 
10 applications such as, for example, medical imaging, biometric verification, signature or 
fingerprint recognition, robot vision, speech recognition, image retrieval, expert systems etc. 

Generally, in decision fusion applications, multiple classifiers (or experts) perform 
separate classification experiments on respective data sets, and consequently designate a 
nominated class as correct. The classifier decisions are then combined in a predetermined 
15 selection strategy to arrive at the final class, as described below. Two extreme approaches for 
the combination strategy are outlined below: 

L The first approach may accept the decision of the majority of the classifiers as the final 

decision (decision consensus approach). 
2. The second approach can take the decision of the most competent expert as the final 
20 decision (most competent expert approach). 

An intermediate approach involves determining a solution in which a consensus 
decision is evaluated in terms of the past track records of the experts. Instead of directly 
accepting the consensus decision, the reliability of each decision is evaluated through various 
kinds of confidence measures. The decision is either accepted or rejected based on the result 
25 of such an evaluation. 

In a further approach, a Bayesian cost function is minimised over all the decisions 
given by the experts. The cost function is defined as the cost of making a wrong decision 
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multiplied by the joint probability of occurrence of the respective decisions. 

None of the above approaches outlined above are rigourously optimal or universally 
applicable, and can be subject to errors or limitations of one kind or another. Accordingly, it 
is an object of the invention to at least attempt to address these and other limitations 
5 associated with the prior art. In particular, it is an object of the invention to generally 
improve the classification accuracy of particular decision fusion applications which rely on 
one of the prior art approaches outlined above. 

Summary of the Invention 

10 The inventive concept is founded in recognition that the reliability of a classifier in a 

decision fusion architecture can vary from sample to sample and from experiment to 
experiment. The inventive concept involves using the decisions from multiple classifiers in a 
decision fusion application to make an informed decision as to the classifier which is likely to 
be correct. 

15 More particularly, the inventive concept resides in a recognition that a strategy of 

assigning confidences to different classifiers in a decision fusion architecture can be used to 
improve the classification accuracy of a decision fusion application. This inventive strategy 
is thought to result in improved classification accuracy. 

Embodiments of the invention involve attempting to optimally adapt the weight 

20 given to a particular classifier from sample to sample, which generally results in improved 
performance compared with prior art approaches. A weight or metric of relative confidence 
is assigned to every classifier, depending upon its sample confidence and overall confidence 
(as subsequently described). For each class, an overall score (or likelihood) is calculated 
which combines individual scores from all classifiers, which allows the class with the highest 

25 score (or likelihood) to be designated as the correct class. 

The invention provides a method suitable for deciding how to classify a sample in 
one of a number of predetermined classes, the method comprising: 
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(a) associating a weight w l} with each of a plurality of classifiers i which are 
class models for how to classify a sample j in one of a number of predetermined classes k; 

(b) calculating for each of said predetermined classes k a weighted summation 
CL jk across said classifiers i of the likelihood l ljk that the sample belongs to that respective 

5 class k, weighted by the weight w y ; and 

(c) designating the sample j as belonging to the class k which has an associated 
weighted summation of likelihoods CL jk which is greatest in value. 

The invention further provides an apparatus for classifying a data sample in one of a 
number of predetermined classes, the apparatus comprising: input means to receive data; and 

10 processor means for calculating associating a weight w y with each of a plurality of classifiers 
i which are class models for how to classify a sample] in one of a number of predetermined 
classes k, and for designating calculating for each of said predetermined classes k a weighted 
summation CL jk across said classifiers i of the likelihood l ljk that the sample belongs to that 
respective class k, weighted by the weight w,j. The weight can be derived from a metric of 

1 5 relative confidence in the decision of a respective classifier i. Preferably, this is an L-statistic 
(linear combination of the order statistic), which represents the statistical separation among 
the order statistic, preferably log-likelihoods, against the class models for a classifier. 

This determination of relative confidence can be performed in two different ways to 
calculate two components of weight given to the decision of a classifier i, referred to as 

20 sample confidence L y and overall confidence H,. Preferably, these confidence values of the 
classifiers i are subsequently used to combine the decisions from the classifiers i to obtain the 
final decision. 

The L-statistic, for a particular sample j, L ljk , can be defined as: 

L y = ai lyi + a 2 lip + + a n l ljn 

25 where l yk denotes for sample j and classifier i, the log-likelihood of the kth most likely class is 
such that the Us form order statistic, that is l y i > i }2 > . - > W The values of a L s define the 
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form of the particular L-statistic L y chosen. Preferably, the order statistic used is simply the 
difference between the log-likelihoods of the two most likely classes k. That is, ai = 1, a 2 = 
-1 and all other a,s = 0. 

A cumulative mean H x of the sample confidence L l} over a large number of samples 
5 is used to measure the overall discrimination capability of the classifier. It is currently 

understood that the value of the 
, overall confidence H, so calculated 

Hi = X_ 1 Wt 

J converges to a constant value which 

is well separated for different overall confidence levels. 
10 Overall confidence for classifier i, H 1; is computed as cumulative mean or moving 

average of the L-statistic L XJ over a number of samples j after which it becomes almost 
constant. 



15 In the equation directly above t is the number of samples after which the overall 

confidence value stabilises. H : attempts to model some kind of disturbance or noise which is 
application specific. Typically, such noise degrades the efficiency of the classifier across all 
classes. For example, in the case of speech recognition, this may be ambient noise (such as 
car noise, cocktail party noise) present in the audio channel There may be, of course, some 

20 cases in which the amount of noise present in the classifier varies during the experiment. 

For every incoming sample j, sample confidence values L 1} $ are computed for every 
classifier i. The overall confidence Ji for the classifiers are updated using L„. Preferably, a 
weight w y is assigned to each classifier i as a function of the overall confidence H L and the 
sample confidence L y . Once weights w 5 for each classifier are known, each incoming sample 

25 j can be classified in a class k by calculating the combined log-likelihood CL jk for each class 
k, as set out directly below. 
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where 



w y = f(L y ,H^ 



For the sample j, the class k with the highest calculated combined log-likelihood 
CLj k is finally chosen as the correct class k for sample j. The l ljk denotes the log likelihood of 
sample j for class k using classifier L 

The invention also includes a computer program product for performing 
embodiments of the inventive methods described above. 

Embodiments of the invention can be used in various applications in which decision 
fusion is conventionally used. 

Brief Description of the Drawings 

Fig. 1 is a schematic representation of the process involved in reaching a decision in 
a classification problem, in accordance with an embodiment of the invention. 

Fig. 2 is a schematic representation of the process involved in determining a weight 
using a threshold value, in accordance with an embodiment of the invention. 

Fig. 3 is a schematic representation of computing hardware suitable for performing 
embodiments of the invention. 

Detailed Description of Embodiments and Best Mode 

An embodiment of the invention is described below in the context of an audiovisual 



there are two relevant classifiers: audio and video. 

In overview, the classification or recognition process initially involves steps as 
outlined in Fig. 1. Initially, in step 10, the process involves associating a metric of relative 




speech recognition application which uses decision 
fusion for classification problems. In this context, 
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confidence with respective classifiers or class models which predict how a sample should be 
recognised. The L y is calculated in step 20 as an L-statistic of the log-likelihoods l yk , as 
detailed below. The moving average H 1? across a suitable number of samples j is then 
determined in step 30. This allows weights to be calculated in step 40 for each classifier 
5 from H and L 1J; according to a suitable function as detailed below. The combined likelihoods 
across classifiers CL jk are then calculated in step 50 as a weighted summation of the 
likelihoods of each class, so that the most likely class can then be determined in step 60. 

For the speech recognition application decision, the problem can be defined as 
follows. Given an audio and a video vector corresponding to a particular speech time frame, 
10 it is necessary to determine the phone class to which this frame belongs. Phones are 
modelled as GMM (Gaussian Mixture Models) obtained from the training data. 

Given an audio vector, its likelihood of corresponding to each of the phone classes is 
computed from the respective classification models. From these likelihoods, the L-statistic is 
preferably chosen simply as the difference between the first and the second most likely 
1 5 choices. As a result, coefficients a x are used as follows. 

ai = 1, a 2 = -1, all other a,s = 0 

A similar computation is also performed for the video vector. The L-statistic is 
shown as L y in Fig. 2. The cumulative mean of L-statistic, H> is used here to model the 
background noise present in the audio channel only, as background noise uniformly degrades 
20 the audio recognition rate across all phonetic sounds. Accordingly, the L-statistic Ly 
decreases uniformly in the presence of noise. The combined likelihood of corresponding to a 
particular phone class is computed as follows. 

CL, = W a *lai + W v *lvi 

Here l ai and 1 V1 are log likelihoods for ith audio class and ith video class respectively, 
25 w a and w v are the weights assigned to the audio and video likelihoods respectively. The 
phone class with the highest combined likelihood is selected as the correct phone class. 
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The weight for audio is determined and, since there are only two classifiers in this 
case, the weight for video is simply determined as the complement of the weight for audio, as 
the linear summation of all weights is 1. A threshold a, is defined for sample confidence 
values of audio which are just the L-statistic in this case. First, the class confidence value for 
5 audio is checked against its threshold in step 100. If it passes this test, audio weight is 
computed in step 110 as a constant term and a term which is dependent on the overall 
confidence of the audio channel. If audio fails this test in step 120, the constant term in the 
weight changes. 

Hence this embodiment, function f() is implemented as 

10 w 1J = f(L 1J ,H l ) = fi(L 1J ) + f 2 (H 1 ) 

where fiO is chosen as a threshold function and f 2 () is given as 

f 2 (H 1 ) = x 1 /(l+exp(x 2 *H 1 )) 

Parameters xi and x 2 are scalar values that are selected and, if necessary, adjusted to 
provide good performance. Preferably, sample confidence is used as a confidence measure 

15 for a classifier for the current sample being processed. The sample confidence models 
non-uniform discrimination capability of the classifier across various classes due to the 
non-uniform dispersion of the clusters in vector space for the data set of the classifier. The 
sample confidence does not represent the overall discrimination capability of the classifier. A 
low value of the sample confidence indicates low confidence in its decision for the present 

20 sample. Similarly, a high value of the sample confidence indicates a higher confidence in its 
decision for that sample. The sample confidence for the present sample is preferably 
represented by the L-statistic for the sample. 

Preferably, overall confidence represents the overall discrimination capability of the 
classifier across all classes (or clusters). This overall discrimination capability may vary 

25 between experiments degraded due to the presence of noise which uniformly degrades the 
classifier's discrimination capability across all classes. For example, in the case of speech 
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recognition, this may be background noise present in the audio channel. 

In this application, it is possible to achieve improvements in phonetic classification 
results using the techniques of the described embodiment of the invention. 

Computer hardware for performing embodiments of the invention is now described. 

5 The described process of classification can be implemented using a computer 

program product in conjunction with a computer system 200 as shown in Fig. 3. In 
particular, the process can be implemented as software, or computer readable program code, 
executing on the computer system 200. 

The computer system 200 includes a computer 250, a video display 210, and input 

10 devices 230, 232. In addition, the computer system 200 can have any of a number of other 
output devices including line printers, laser printers, plotters, and other reproduction devices 
connected to the computer 250. The computer system 200 can be connected to one or more 
other computers via a communication input/output (I/O) interface 264 using an appropriate 
communication channel 240 such as a modem communications path, an electronic network, 

15 or the like. The network may include a local area network (LAN), a wide area network 
(WAN), an Intranet, and/or the Internet 220. 

The computer 250 includes the control module 266, a memory 270 that may include 
random access memory (RAM) and read-only memory (ROM), input/output (I/O) interfaces 
264, 272, a video interface 260, and one or more storage devices generally represented by the 

20 storage device 262. The control module 266 is implemented using a central processing unit 
(CPU) that executes or runs a computer readable program code that performs a particular 
function or related set of functions. 

The video interface 260 is connected to the video display 210 and provides video 
signals from the computer 250 for display on the video display 210. User input to operate the 

25 computer 250 can be provided by one or more of the input devices 230, 232 via the I/O 
interface 272. For example, a user of the computer 250 can use a keyboard as I/O interface 
230 and/or a pointing device such as a mouse as I/O interface 232. The keyboard and the 
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mouse provide input to the computer 250. The storage device 262 can consist of one or more 
of the following: a floppy disk, a hard disk drive, a magneto-optical disk drive, CD-ROM, 
magnetic tape or any other of a number of non-volatile storage devices well known to those 
skilled in the art. Each of the elements in the computer system 250 is typically connected to 

5 other devices via a bus 280 that in turn can consist of data, address, and control buses. 

The method steps for are effected by instructions in the software that are carried out 
by the computer system 200. Again, the software may be implemented as one or more 
modules for implementing the method steps. 

In particular, the software may be stored in a computer readable medium, including 

10 the storage device 262 or that is downloaded from a remote location via the interface 264 and 
communications channel 240 from the Internet 220 or another network location or site. The 
computer system 200 includes the computer readable medium having such software or 
program code recorded such that instructions of the software or the program code can be 
carried out. The use of the computer system 200 preferably effects advantageous apparatuses 

15 for constructing a runtime symbol table for a computer program in accordance with the 
embodiments of the invention. 

The computer system 200 is provided for illustrative purposes and other 
configurations can be employed without departing from the scope and spirit of the invention. 
The foregoing is merely an example of the types of computers or computer systems with 

20 which the embodiments of the invention may be practised. Typically, the processes of the 
embodiments are resident as software or a computer readable program code recorded on a 
hard disk drive as the computer readable medium, and read and controlled using the control 
module 266. Intermediate storage of the program code and any data including entities, 
tickets, and the like may be accomplished using the memory 270, possibly in concert with the 

25 storage device 262. 

In some instances, the program may be supplied to the user encoded on a CD-ROM 
or a floppy disk (both generally depicted by the storage device 262), or alternatively could be 
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read by the user from the network via a modem device connected to the computer 250. Still 
further, the computer system 200 can load the software from other computer readable media. 
This may include magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio 
or infra-red transmission channel between the computer and another device, a computer 

5 readable card such as a PCMCIA card, and the Internet 220 and Intranets including email 
transmissions and information recorded on Internet sites and the like. The foregoing are 
merely examples of relevant computer readable media. Other computer readable media may 
be practised without departing from the scope and spirit of the invention. 

Further to the above, the described methods can be realised in a centralised fashion 

10 in one computer system 200, or in a distributed fashion where different elements are spread 
across several interconnected computer systems. 

Computer program means or computer program in the present context mean any 
expression, in any language, code or notation, of a set of instructions intended to cause a 
system having an information processing capability to perform a particular function either 

15 directly or after either or both of the following: a) conversion to another language, code or 
notation or b) reproduction in a different material form. 

In the foregoing manner, a method, an apparatus, and a computer program product 
for are disclosed. While only a small number of embodiments are described, it will be 
apparent to those skilled in the art in view of this disclosure that numerous changes and/or 

20 modifications can be made without departing from the scope and spirit of the invention. 

It is understood that the invention is not limited to the embodiment described, but 
that various alterations and modifications, as would be apparent to one skilled in the art, are 
included within the scope of the invention. 
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We claim: 



1 . A method suitable for deciding how to classify a sample in one of a number 
5 of predetermined classes, the method comprising: 

(a) associating a weight w y with each of a plurality of classifiers i which are 
class models for how to classify a sample j in one of a number of predetermined classes k; 

(b) calculating for each of said predetermined classes k a weighted summation 
CL jk across said classifiers i of the likelihood l 1Jk that the sample belongs to that respective 

1 0 class k, weighted by the weight w u ; and 

(c) designating the sample j as belonging to the class k which has an associated 
weighted summation of likelihoods CL jk which is greatest in value. 

2. The method as claimed in claim 1, wherein the weight w y is derived from a 
15 metric of relative confidence L y , metric of relative which is calculated as an L-statistic, or 

linear combination of an order statistic, which represents the statistical separation among an 
order statistic of the classes k for a particular classifier i. 

3. The method as claimed in claim 2, wherein the L-statistic L y is of the 
20 log-likelihoods of respective classes k for classifiers i. 

4. The method as claimed in claim 2, wherein the L-statistic L y , for a 
particular sample j, is calculated as: L, = a, l yl + a 2 l lj2 + ... + a„ l Ijn , where l yk s form order 
statistic, that is l yl > l lj2 > . . . > U and a; = 1, a 2 = -1 and all other a,s = 0. 



25 



5. The method as claimed in claim 2, wherein the weight w, derived from the 
metric of relative confidence is calculated as a function of (a) sample confidence L 1} , equal to 
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the L-statistic L y and (b) overall confidence H 1? the cumulative mean of the sample confidence 
L y over a plurality of samples j. 

6. The method as claimed in claim 5, wherein the overall confidence H t is 
5 successively updated with the sample confidence L y of each sample j . 

7. A computer program product having a computer readable medium 
having a computer program recorded therein for deciding how to classify a 
sample in one of a number of predetermined classes, said computer 

1 0 program product comprising: 

(a) code means for associating a weight w y with each of a plurality of 
classifiers i which are class models for how to classify a sample j in one of a number of 
predetermined classes k; 

(b) code means for calculating for each of said predetermined classes k a 
15 weighted summation CL jk across said classifiers i of the likelihood l 1Jk that the sample belongs 

to that respective class k, weighted by the weight w y ; and 

(c) code means designating the sample j as belonging to the class k which has 
an associated weighted summation of likelihoods CL jk which is greatest in value. 

20 8. An apparatus for classifying a data sample in one of a number of 

predetermined classes, the apparatus comprising: input means to receive data; 
and processor means for calculating associating a weight w y with each of a 
plurality of classifiers i which are class models for how to classify a sample j 
in one of a number of predetermined classes k, and for designating calculating 

25 for each of said predetermined classes k a weighted summation CL Jk across 

said classifiers i of the likelihood l 1Jk that the sample belongs to that respective 
class k, weighted by the weight Wy 
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9. The apparatus as claimed in claim 8, wherein the weight w u is derived 
from a metric of relative confidence L* metric of relative which is calculated 
as an L-static, or linear combination of an order statistic, which represents the 
statistical separation among an order statistic of the classes k for a particular 
classifier i. 

10. The apparatus as claimed in claim 9, wherein the L-statistic L y is of 
the log-likelihoods of respective classes k for classifiers i. 

11 The apparatus as claimed in claim 9, wherein the L-statistic L XJ , for a 

particular j, is calculated as: L y = en lyi + a 2 l lj2 + ... + a n l ljn , where l yk s form 
order statistic, that is l y i > l lj2 > . . . > lyn and * = 1, a 2 = -1 and all other a>s = 0. 

12. The apparatus as claimed in claim 9, wherein the weight w, derived 
from the metric of relative confidence is calculated as a function of (a) sample 
confidence L y , equal to the L-statistic L y and (b) overall confidence H„ the 
cumulative mean of the sample confidence L y over a plurality of samples j. 

13. The apparatus as claimed in claim 12, wherein, the overall confidence 
H a is successively updated with the sample confidence L y of each sample j . 
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ABSTRACT 

A method of classifying samples to one of a number of predetermined classes 
involves using a number of class models or classifiers to form order statistic for each 
5 classifier. A linear combination of the order statistic (L-statistic) is calculated to determine 
the confidence of that particular classifier, both in general and for that particular sample. 
Relative weights are then derived from these confidences, and used to calculate a weighted 
summation across all classifiers for each class of the likelihoods that a sample belongs to that 
class. The sample is classified in the class which has the associated weighted summation 
1 0 which i s greatest in value. 
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application (s) for patent or inventor's certificate listed below and have also identified below any foreign 
application for patent or inventor's certificate having a filing date before that of the application on which 
priority is claimed: 

„ -> ■ , v Priority Claimed 

Prior Foreign Application ( s ) J 

Yes No 



(Number) 


(Country) 


(Day/Month/ Year Filed) 


(Number) 


(Country) 


(Day/Month/Year Filed) 


(Number) 


(Country) 


(Day/Month/Year Filed} 



Yes No 

Yes No 



I hereby claim the benefit under 35 U.S.C. 5119(e) of any United States provisional application ( s ) listed 
below. 



(Application Number) (Filing Date) 



(Application Number) (Filing Date) 

I hereby claim the benefit under Title 35, United States Code, Section 120 of any United States 
Application (s) listed below and, Insofar as the subject matter of each of the claims of the application is 
not disclosed in the prior United States application in the manner provided by the first paragraph of Title 
35, United States Code, Section 112, I acknowledge the duty to disclose information material to the 
patentability of this application as defined in Title 37, Code of Federal Regulations, Section 1 56 which 
occurred between the filing date of the prior application and the national or PCT international filing date 
of this application; 



(Applica t ion Serial No.) (Filing Date) (Status) (patented, pending, abandoned) 



(Application Serial No.) (Filing Date) (Status) (patented, pending, abandoned) 

I hereby declare that all statements made herein of my own knowledge are true and that all statements made on 
information and belief are believed to be true;^and further that these statements were made with the 
knowledge that willful false statements and the like so made are punishable by fine or imprisonment, or both, 
under Section 1001 of Title 18 of the United States Code and that willful false statements may jeopardize the 
validity of the application or any patent issued thereon. 

POWER OF ATTORNEY: As a named inventor I hereby appoint the following attorney (s) and/or agent (s) to 
prosecute this application and transact all business m the Patent and Trademark Office connected therewith 
(list name and registration number) . 

Manny W. Schecter (Reg. 31,722), Terry J. Ilardi (Reg. 29,936), Christopher A. Hughes (Reg 26,914), Edward 
A Pennington (Reg. 32,588), John E . Hoel (Reg. 26,279), Joseph C. Redmond, Jr. (Reg. 18,753), Kevin M 
Jordan (Reg. 40,277), Stephen C. Kaufman (Reg. 29,551), Jay. P. Sbrollini (Reg. 36, 266), David M. Shofi 



Reg. 39,835), Robert M. Trepp (Reg. 25,933), Louis P. Herzberg (Reg. 41,500), and Douglas W. Cameron (Reg. 
1,596), Paul Otterstedt (Reg. 37,411), Louis J. Percello (Reg. 33,206) and Daniel P. Morris (Reg. 32,053). 
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Express Mail EL627131289US 
Date of Deposit: Apr. 20, 2001 



Send Correspondence to: Stephen C. Kaufman 

IBM Corporation (39-239) 
PO Box 218 

Route 134 Kitchawan Road, Yorktown Heights, NY 10598 

Direct Telephone Calls to: , _ rt „ , ^ , ^ 

Stephen C. Kaufman, 914-945-3197 



Full name of original, first and joint inventor Ashish Verma 

Inventor ' s Signature ^^^^^^T^^ Date 

Residence A-92, Shivalik Colony, New Delhi - 110 017 

Citizenship India 

post office Address A-92, Shivalik Colony, New Delhi - 110 017, India 
Full name of original, first and joint inventor Abhinanda Sarkar 



Inventor's Signature 



Date 2-C/62,|0&Pl 



Residence N-86, Panchsheel Park, New Delhi - 110 017 

Citizenship India 

Post Office Address N-86, Panchsheel Park, New Delhi - 110 017, India 
Full name of original, first and joint inventor Arpita Ghosh 

inventor's Signature Date £?> MMW %OQ\ 



Residence 7-B, Annapurna, Anushakti Nagar, Mumbai - 400 094 

Citizenship India 

Post Office Address 7-B, Annapurna, Anushakti Nagar, Mumbai -4 00 094, India 
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